The goal of super-resolution methods is to recover a high resolution signal from one or more low resolution input signals. Methods for super-resolution of images can be broadly classified into two families of methods: the classical multi-image super-resolution and example-based super-resolution.
Reference is now made to FIG. 1, which generally illustrates the two types of methods.
In classical multi-image super-resolution such as described in Michal Irani and Shmuel Peleg, “Improving resolution by image registration”, CVGIP: Graphical Model and Image Processing, Vol. 53, (3), 1991, pp 231-239, Capel, David, “Image Mosaicing and Super-Resolution”, Springer Verlag, 2004, and Farsiu, S., Robinson, M D, Elad, M. and Milanfar, P., “Fast and robust multiframe super resolution”, IEEE Transactions on Image Processing, Vol. 13, (10), 2004, pp 1327-1344, a set of low-resolution images Li of the same scene are taken, at sub pixel misalignments, and the method attempts to generate a high-resolution image H from the multiple low-resolution images Li.
Each pixel p of the low resolution image Lj is assumed to be generated from high resolution image H by a blurring function Bj and a sub-sampling process sj, as given by Equation 1.Lj=(H*Bj)↓sj  Equation 1
Thus, each low resolution pixel p in each low resolution image induces one linear constraint on the unknown high-resolution intensity values within a local neighborhood R around its corresponding high-resolution pixel q, where the size of the neighborhood is determined by the “support” of the blur kernel Bj:
                                          L            j                    ⁡                      (            p            )                          =                                            (                              H                *                                  B                  j                                            )                        ⁢                          (              q              )                                =                                    ∑                                                q                  i                                ∈                                  Support                  ⁢                                                                          ⁢                                      (                                          B                      j                                        )                                                                        ⁢                                          H                ⁡                                  (                                      q                    i                                    )                                            ⁢                                                B                  j                                ⁡                                  (                                                            q                      i                                        -                    q                                    )                                                                                        Equation        ⁢                                  ⁢        2            
where H(qi) are the unknown high-resolution intensity values and Bj is the blur kernel relating high resolution image H with low resolution image LJ. Thus, region Ri(p), the region which affects pixel p in image Lj, is a function of p, the alignment between Lj and H at sub-pixel accuracy and the blur kernel Bj. A blurring function may be a point spread function and the amount of blur typically is a function of the extent of the sub-sampling, where the more the sub-sampling, the larger the blur.
Thus, each pixel Li(p) of low resolution image Li imposes a constraint on local region Ri(p). If enough low-resolution images are available (at subpixel shifts from each other), then the set of equations becomes determined and can be solved to recover the high-resolution image.
Such super-resolution schemes have been shown to provide reasonably stable results up to a factor of about 2, but are limited in the presence of noise and misregistration. These limitations have lead to the development of “Example-Based Super-Resolution” introduced in articles by Freeman, W. T. and Pasztor, E. C. and Carmichael, O. T., “Learning low-level vision”, International Journal of Computer Vision, Vol. 40 (1), 2000, Springer, pp 25-47, Freeman, William T., Jones, Thouis R. and Pasztor, Egon C., “Example-Based Super-Resolution”, IEEE Comput. Graph. Appl., Vol. 22, March 2002, (2), pp. 56-65, and Simon Baker and Takeo Kanade, “Limits on super-resolution and how to break them”, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2002, Vol. 24, pp. 1167-1183 and extended later by others.
In example-based super-resolution, pairs of corresponding images L and H (generally with a relative scale factor of 2) are stored in a database 10, where example images LEi are of low-resolution and example images HE are their high resolution version. It is also common to store examples which have a scale gap which is the same as the scale gap between the desired high resolution image and the input low-resolution image. For each patch PT in the input low resolution image L, a similar patch LE(PT) is found in one of the low-resolution example images LEi in database 10. A corresponding high-resolution patch HE(PT) is then copied to the unknown high-res image H at the appropriate location.
Example-based super-resolution thus utilizes the known correspondences between low resolution and high resolution image patches in database 10. Higher super-resolution factors have often been obtained by repeated applications of this process.
It is important to distinguish between image interpolation (or image zooming) and super-resolution. In image interpolation, the goal is to magnify (scale-up) an image while maintaining the sharpness of the edges and the details in the image. In contrast, in super-resolution the goal is to recover new (i.e. missing) high-resolution details beyond the Nyquist frequency of the low-resolution image that are not explicitly found in any individual low-resolution image. In classical super-resolution, this high-frequency information is assumed to be split across multiple low-resolution images, implicitly found there in aliased form. In example-based super-resolution, this missing high-resolution information is assumed to be available in the high-resolution database patches, and is learned from the low-resolution/high-resolution pairs of examples in the database.